Methods and systems for performing computer assisted surgery

ABSTRACT

Methods and systems for performing computer-assisted surgery, including robot-assisted image-guided surgery. Embodiments include marker devices for an image guided surgery system, marker systems and arrays for tracking a robotic arm using a motion tracking system, and image guided surgery methods and systems using optical sensors.

RELATED APPLICATIONS

This application claims the benefit of priority to U.S. ProvisionalApplication No. 62/568,354, filed on Oct. 5, 2017, the entire contentsof which are incorporated herein by reference.

BACKGROUND

Computer-assisted surgical procedures, which may include image guidedsurgery and robotic surgery, have attracted increased interest in recentyears. These procedures include the integration of a “virtual”three-dimensional dataset of the patient's anatomy, typically obtainedusing pre-operative or intra-operative medical imaging (e.g., x-raycomputed tomography (CT) or magnetic resonance (MR) imaging), to theactual position of the patient and/or other objects (e.g., surgicalinstruments, robotic manipulator(s) or end effector(s) in the surgicalarea. These procedures may be used to aid the surgeon in planning asurgical procedure and may also provide the surgeon with relevantfeedback during the course of surgical procedure. There is a continuingneed to improve the safety and ease-of-use of computer-assisted surgicalsystems.

SUMMARY

Various embodiments include methods and systems for performingcomputer-assisted surgery, including robot-assisted image-guidedsurgery.

Embodiments include a marker device for an image guided surgery systemthat includes an electronics unit having at least one light source, arigid frame attached to the electronics unit, the rigid frame having atleast one channel extending from the electronics unit to at least oneopening in the rigid frame, and an optical guide apparatus locatedwithin the at least one channel to couple light from the at least onelight source of the electronics unit to the at least one opening in therigid frame.

Further embodiments include a marker device for an image guided surgerysystem that includes an electronics unit including a flexible circuithaving a plurality of peripheral arm regions and a light source locatedon each of the peripheral arm regions, and a rigid frame attached to theelectronics unit, the rigid frame having a plurality of channelsterminating in openings in the rigid frame, each of the plurality ofperipheral arm regions located within a channel with each of theplurality of light sources configured to direct light from a respectiveopening in the rigid frame.

Further embodiments include a marker system for tracking a robotic armusing a motion tracking system that includes a light source locatedwithin the robotic arm, and a marker comprising an optical diffuser thatattaches to an outer surface of the robotic arm to optically couple thelight source to the diffuser.

Further embodiments include a marker array having a plurality of markersfor tracking a robotic arm that includes multiple axes between aproximal end and a distal end of the robotic arm, and an end effectorattached to the distal end of the robotic arm, where the marker arrayincludes at least one first marker that is distal to the most distalaxis of the robotic arm, and at least one second marker that is proximalto the most distal axis of the robotic arm.

Further embodiments include a multi-axis robotic arm that includes afirst section that comprises at least one axis that provides both pitchand yaw rotation, a second section, distal to the first section, thatcomprises two mutually orthogonal rotary wrist axes, and an end effectorcoupled to the second section.

Further embodiments include an image guided surgery system that includesan optical sensor facing in a first direction to detect optical signalsfrom a marker device located in a surgical site, a reference markerdevice located along a second direction with respect to the opticalsensor, and a beam splitter optically coupled to the optical sensor andconfigured to redirect optical signals from the reference marker deviceto the optical sensor.

Further embodiments include an optical sensing device for a motiontracking system that includes a support structure, at least one opticalsensor mounted to the support structure and configured to generatetracking data of one or more objects within a field-of-view of theoptical sensor, and an inertial measurement unit mounted to the supportstructure and configured to detect a movement of the at least oneoptical sensor.

Further embodiments include an image guided surgery system that includesa marker device, a least one optical sensor configured to detect opticalsignals from the marker device, an inertial measurement unitmechanically coupled to the at least one optical sensor and configuredto measure a movement of the at least one optical sensor, and aprocessing system, coupled to the at least one optical sensor and theinertial measurement unit, and including at least one processorconfigured with processor-executable instructions to perform operationsincluding tracking the position and orientation of the marker devicebased on the optical signals received at the at least one opticalsensor, receiving measurement data from the inertial measurement unitindicating a movement of the at least one optical sensor, and correctingthe tracked position and orientation of the marker device based on themeasurement data from the inertial measurement unit.

Further embodiments include a method of performing image guided surgerythat includes tracking the position and orientation of a marker devicebased on optical signals from the marker device received by at least oneoptical sensor, receiving measurement data from an inertial measurementunit indicating a movement of the at least one optical sensor, andcorrecting the tracked position and orientation of the marker devicebased on the measurement data from the inertial measurement unit.

Further embodiments include an image guided robotic surgery system thatincludes a robotic arm, a plurality of marker devices, a sensor arraylocated on the robotic arm and configured to detect optical signals fromthe plurality of marker devices, and a processing system, coupled to thesensor array, and configured to track the position of the plurality ofmarker devices in three-dimensional space based on the detected opticalsignals from the sensor array and the joint coordinates of the roboticarm.

BRIEF DESCRIPTION OF THE DRAWINGS

Other features and advantages of the present invention will be apparentfrom the following detailed description of the invention, taken inconjunction with the accompanying drawings of which:

FIG. 1 is a perspective view of a system for performingrobotically-assisted image-guided surgery according to an embodiment.

FIG. 2 shows an alternative embodiment of a system for performingrobotically-assisted image-guided surgery having an optical sensingdevice for a motion tracking system on an arm extending from a gantry ofan imaging system.

FIG. 3 is a process flow diagram illustrating a method for performingregistration of patient image data for image-guided surgery.

FIG. 4 is a block diagram schematically illustrating a system forrobotically-assisted image-guided surgery according to an embodiment.

FIG. 5 illustrates a display screen of a display device in animage-guided surgery system according to an embodiment.

FIGS. 6A-6C schematically illustrate an image guided surgical systemthat includes an optical system for providing a visible indication of arange of a motion tracking system.

FIG. 7A schematically illustrates an optical sensing device for a motiontracking system that includes at least one beam splitter coupled to theoptical sensing device for tracking markers located in multipledirections.

FIG. 7B schematically illustrates an optical sensing device for a motiontracking system that includes an inertial measurement unit attached toan optical sensing device to correct for movements of the opticalsensing device.

FIGS. 8A-8E illustrate robotic arms, including marker devices attachedto a robotic arm.

FIGS. 9A-9B illustrate marker devices on a robotic arm.

FIGS. 10A-10D illustrate various embodiments of an active marker devicefor a motion tracking system.

FIGS. 11A-11B illustrate a motion tracking system for robot-assistedimaged guided surgery that includes motion tracking sensors located on arobotic arm.

FIG. 12 schematically illustrates a computing device which may be usedfor performing various embodiments.

DETAILED DESCRIPTION

The various embodiments will be described in detail with reference tothe accompanying drawings. Wherever possible, the same reference numberswill be used throughout the drawings to refer to the same or like parts.References made to particular examples and implementations are forillustrative purposes, and are not intended to limit the scope of theinvention or the claims.

FIG. 1 illustrates a system 100 for performing computer assistedsurgery, including robotically-assisted image-guided surgery accordingto various embodiments. The system 100 in this embodiment includes animaging device 103, a motion tracking system 105 and a robotic arm 101for performing a robotically-assisted surgical procedure. The roboticarm 101 may comprise a multi joint arm that includes a plurality oflinkages connected by joints having actuator(s) and optional encoder(s)to enable the linkages to rotate, bend and/or translate relative to oneanother in response to control signals from a robot control system. Therobotic arm 101 may be fixed to a support structure at one end and mayhave an end effector 102 at the other end of the robotic arm 101.

The imaging device 103 may be used to obtain diagnostic images of apatient 200, which may be a human or animal patient. In embodiments, theimaging device 103 may be an x-ray computed tomography (CT) imagingdevice. The patient 200 may be positioned within a central bore 107 ofthe imaging device 103 and an x-ray source and detector may be rotatedaround the bore 107 to obtain x-ray image data (e.g., raw x-rayprojection data) of the patient 200. The collected image data may beprocessed using a suitable processor (e.g., computer) to perform athree-dimensional reconstruction of the object. In other embodiments,the imaging device 103 may comprise one or more of an x-ray fluoroscopicimaging device, a magnetic resonance (MR) imaging device, a positronemission tomography (PET) imaging device, a single-photon emissioncomputed tomography (SPECT), or an ultrasound imaging device. Inembodiments, image data may be obtained pre-operatively (i.e., prior toperforming a surgical procedure) or intra-operatively (i.e., during asurgical procedure) by positioning the patient 200 within the bore 107of the imaging device 103. In the system 100 of FIG. 1 , this may beaccomplished by moving the imaging device 103 over the patient 200 toperform a scan while the patient 200 may remain stationary.

Examples of x-ray CT imaging devices that may be used according tovarious embodiments are described in, for example, U.S. Pat. No.8,118,488, U.S. Patent Application Publication No. 2014/0139215, U.S.Patent Application Publication No. 2014/0003572, U.S. Patent ApplicationPublication No. 2014/0265182 and U.S. Patent Application Publication No.2014/0275953, the entire contents of all of which are incorporatedherein by reference. In the embodiment shown in FIG. 1 , the patientsupport 60 (e.g., surgical table) upon which the patient 200 may belocated is secured to the imaging device 103, such as via a column 50which is mounted to a base 20 of the imaging device 103. A portion ofthe imaging device 103 (e.g., an O-shaped imaging gantry 40) whichincludes at least one imaging component may translate along the lengthof the base 20 on rails 23 to perform an imaging scan of the patient200, and may translate away from the patient 200 to an out-of-the-wayposition for performing a surgical procedure on the patient 200.

An example imaging device 103 that may be used in various embodiments isthe AIRO® intra-operative CT system manufactured by Mobius Imaging, LLCand distributed by Brainlab, AG. Other imaging devices may also beutilized. For example, the imaging device 103 may be a mobile CT devicethat is not attached to the patient support 60 and may be wheeled orotherwise moved over the patient 200 and the support 60 to perform ascan. Examples of mobile CT devices include the BodyTom® CT scanner fromSamsung Electronics Co., Ltd. and the O-Arm® surgical imaging systemform Medtronic, plc. The imaging device 103 may also be a C-arm x-rayfluoroscopy device. In other embodiments, the imaging device 103 may bea fixed-bore imaging device, and the patient 200 may be moved into thebore of the device, either on a surgical support 60 as shown in FIG. 1 ,or on a separate patient table that is configured to slide in and out ofthe bore. Further, although the imaging device 103 shown in FIG. 1 islocated close to the patient 200 within the surgical theater, theimaging device 103 may be located remote from the surgical theater, suchas in another room or building (e.g., in a hospital radiologydepartment).

The motion tracking system 105 shown in FIG. 1 includes a plurality ofmarker devices 119, 202, 115 and an optical sensor device 111. Varioussystems and technologies exist for tracking the position (includinglocation and/or orientation) of objects as they move within athree-dimensional space. Such systems may include a plurality of activeor passive markers fixed to the object(s) to be tracked and a sensingdevice that detects radiation emitted by or reflected from the markers.A 3D model of the space may be constructed in software based on thesignals detected by the sensing device.

The motion tracking system 105 in the embodiment of FIG. 1 includes aplurality of marker devices 119, 202 and 115 and a stereoscopic opticalsensor device 111 that includes two or more cameras (e.g., IR cameras).The optical sensor device 111 may include one or more radiation sources(e.g., diode ring(s)) that direct radiation (e.g., IR radiation) intothe surgical field, where the radiation may be reflected by the markerdevices 119, 202 and 115 and received by the cameras. The marker devices119, 202, 115 may each include three or more (e.g., four) reflectingspheres, which the motion tracking system 105 may use to construct acoordinate system for each of the marker devices 119, 202 and 115. Acomputer 113 may be coupled to the sensor device 111 and may determinethe transformations between each of the marker devices 119, 202, 115 andthe cameras using, for example, triangulation techniques. A 3D model ofthe surgical space in a common coordinate system may be generated andcontinually updated using motion tracking software implemented by thecomputer 113. In embodiments, the computer 113 may also receive imagedata from the imaging device 103 and may register the image data to thecommon coordinate system as the motion tracking system 105 using imageregistration techniques as are known in the art. In embodiments, areference marker device 115 (e.g., reference arc) may be rigidlyattached to a landmark in the anatomical region of interest (e.g.,clamped or otherwise attached to a bony portion of the patient'sanatomy) to enable the anatomical region of interest to be continuallytracked by the motion tracking system 105. Additional marker devices 119may be attached to surgical tools 104 to enable the tools 104 to betracked within the common coordinate system. Another marker device 202may be rigidly attached to the robotic arm 101, such as on the endeffector 102 of the robotic arm 101, to enable the position of roboticarm 101 and end effector 102 to be tracked using the motion trackingsystem 105. The computer 113 may also include software configured toperform a transform between the joint coordinates of the robotic arm 101and the common coordinate system of the motion tracking system 105,which may enable the position and orientation of the end effector 102 ofthe robotic arm 101 to be controlled with respect to the patient 200.

In addition to passive marker devices described above, the motiontracking system 105 may alternately utilize active marker devices thatmay include radiation emitters (e.g., LEDs) that may emit radiation thatis detected by an optical sensor device 111. Each active marker deviceor sets of active marker devices attached to a particular object mayemit radiation in a pre-determined strobe pattern (e.g., with modulatedpulse width, pulse rate, time slot and/or amplitude) and/or wavelengthwhich may enable different objects to be uniquely identified and trackedby the motion tracking system 105. One or more active marker devices maybe fixed relative to the patient, such as secured to the patient's skinvia an adhesive membrane or mask. Additional active marker devices maybe fixed to surgical tools 104 and/or to the end effector 102 of therobotic arm 101 to allow these objects to be tracked relative to thepatient.

In further embodiments, the marker devices may be passive maker devicesthat include moiré patterns that may enable their position andorientation to be tracked in three-dimensional space using a singlecamera using Moiré Phase Tracking (MPT) technology. Each moiré patternmarker may also include a unique identifier or code that may enabledifferent objects within the camera's field of view to be uniquelyidentified and tracked. An example of an MPT-based tracking system isavailable from Metria Innovation Inc. of Milwaukee, Wis. Other trackingtechnologies, such as computer vision systems and/or magnetic-basedtracking systems, may also be utilized.

The system 100 may also include a display device 121 as schematicallyillustrated in FIG. 1 . The display device 121 may display image data ofthe patient's anatomy obtained by the imaging device 103. The displaydevice 121 may facilitate planning for a surgical procedure, such as byenabling a surgeon to define one or more target positions in thepatient's body and/or a path or trajectory into the patient's body forinserting surgical tool(s) to reach a target position while minimizingdamage to other tissue or organs of the patient. The position and/ororientation of one or more objects tracked by the motion tracking system105 may be shown on the display 121, and may be shown overlaying theimage data. In the embodiment of FIG. 1 , the display 121 is located ona mobile cart 120. A computer 113 for controlling the operation of thedisplay 121 may also be housed within the cart 120. In embodiments, thecomputer 113 may be coupled to the optical sensor device 111 and mayalso perform all or a portion of the processing (e.g., trackingcalculations) for the motion tracking system 105. Alternatively, one ormore separate computers may perform the motion tracking processing, andmay send tracking data to computer 113 on the cart 120 via a wired orwireless communication link. The one or more separate computers for themotion tracking system 105 may be located on the imaging system 103, forexample.

FIG. 2 illustrates an alternative embodiment of a system for performingrobotically-assisted image-guided surgery according to variousembodiments. In this embodiment, the optical sensor device 111 includesan array of cameras 207 mounted to a rigid support 208. The support 208including the camera array is suspended above the patient surgical areaby an arm 209. The arm 209 may be mounted to or above the imaging device103. The position and/or orientation of the rigid support 208 may beadjustable with respect to the arm 209 to provide the camera array witha clear view into the surgical field while avoiding instructions. Inembodiments, the rigid support 208 may pivot with respect to the arm 209via a joint 213. In some embodiments, the position of the rigid support208 may be adjustable along the length of the arm 209. A handle 210attached to the rigid support 208 may be used to adjust the orientationand/or position of the optical sensor device 111. The optical sensordevice 111 may be normally locked in place with respect to the arm 209during an imaging scan or surgical procedure. A release mechanism 213 onthe handle 210 may be used to unlock the optical sensor device 111 toenable its position and/or orientation to be adjusted by the user. Insome embodiments, the arm 209 or a portion thereof may be pivotable withrespect to the imaging system 103, such as via joint 214. Inembodiments, the arm 209 may be raised or lowered relative to the topsurface of the imaging system 103, and in some embodiments the entirearm 208 may reciprocate (e.g., to the left or right in FIG. 2 ) withrespect to the imaging system 103.

In some embodiments, the rigid support 208 and cameras 207 may beremovably secured to the arm 209 so that the support 208 and cameras 207may be detached from the system for storage and/or transport. A dockingsystem between the arm 209 and the rigid support 208 may providemechanical coupling between the support 208 and the arm 209 and may alsoprovide an electrical connection for data and/or power between the arm209 and the array of cameras 207 mounted to the support 208.

FIG. 2 also illustrates a display device that may comprise a handhelddisplay device 401. As used herein, “handheld computing device” and“handheld display device” are used interchangeably to refer to any oneor all of tablet computers, smartphones, pendant controllers, cellulartelephones, personal digital assistants (PDA's), netbooks, e-readers,laptop computers, palm-top computers, wearable computers, and similarportable electronic devices which include a programmable processor andmemory coupled to a display screen and may include hardware and/orsoftware to enable display of information, including patient informationand/or images, on the display screen. A handheld computing devicetypically also includes an antenna coupled to circuitry (e.g., atransceiver) to enable wireless communication over a network. A handheldcomputing or display device may be characterized by a sufficientlycompact and lightweight structure to enable a user to easily grasp,maneuver and operate the device using one or both hands.

A holder for a handheld computing device 401 may be in a suitablelocation to enable the user to easily see and/or interact with thedisplay screen and to grasp and manipulate the handheld computing device401. The holder may be a separate cart or a mount for the handheldcomputing device that may be attached to the patient support 60 orcolumn 50 or to any portion of the imaging system 103, or to any of thewall, ceiling or floor in the operating room. In some embodiments, ahandheld computing device 401 may be suspended from the arm 209 to whichthe optical sensing device 111 is attached. One or more handheld displaydevices 401 may be used in addition to or as an alternative to aconventional display device, such as a cart-mounted monitor displaydevice 121 as shown in FIG. 1 .

As shown in FIGS. 1 and 2 , the robotic arm 101 may be fixed to theimaging device 103, such as on a support element 215 (e.g., a curvedrail) that may extend concentrically over the outer surface of theO-shaped gantry 40 of the imaging device 103. In embodiments, an arm 209to which the optical sensing device 111 is mounted may be mounted to thesame or a similar support element 215 (e.g., curved rail) as the roboticarm 101. The position of the robotic arm 101 and/or the support arm 209may be adjustable along the length of the support element 215. In otherembodiments, the robotic arm 101 may be secured to any other portion ofthe imaging device 103, such as directly mounted to the gantry 40.Alternatively, the robotic arm 101 may be mounted to the patient support60 or column 50, to any of the wall, ceiling or floor in the operatingroom, or to a separate cart. In further embodiments, the robotic arm 101and/or the optical sensing device 111 may be mounted to a separatemobile shuttle, as described in U.S. patent application Ser. No.15/706,210, filed on Sep. 15, 2017, which is incorporated by referenceherein. Although a single robotic arm 101 is shown in FIGS. 1 and 2 , itwill be understood that two or more robotic arms 101 may be utilized.

FIG. 3 is a process flow diagram that illustrates a method 300 ofregistering patient images. Computer-assisted surgery techniquesgenerally utilize a process of correlating a dataset representing aportion of the patient's anatomy that is to be operated on with theposition of the patient at the time of the surgical intervention. Theposition of the patient may be determined based on a second imagedataset which may include real-time camera image(s) from a motiontracking system 105 as described above. The correlation between thesedatasets may be accomplished computationally using software, and may bereferred to as “patient registration.” The registration method 300 ofFIG. 3 may be implemented using one or more computing devices, such ascomputer 113 shown in FIG. 1 .

In block 301 of method 300, a first image dataset of the patient'sanatomy may be obtained using an imaging device, such as the imagingdevice 103 shown in FIGS. 1 and 2 . The first image dataset may be athree-dimensional dataset (e.g., a 3D CT tomographic reconstruction, a3D MRI dataset, etc.) representing at least a portion of the patient'sanatomy, including the internal anatomy and/or structure(s) that are tobe operated on (i.e., a surgically-relevant portion of the patient'sanatomy). The first image dataset may be stored electronically in amemory. The first image dataset may be in any suitable format, such asin a file format that conforms to the Digital Imaging and Communicationsin Medicine (DICOM) standard.

In block 303 of method 300, a second image dataset of the patient andthe surrounding patient space may be obtained using a motion trackingsystem, such as the motion tracking system 105 shown in FIGS. 1 and 2 .The second image dataset may indicate the current position and/ororientation of the patient. The second image dataset may include atleast one image of a marker device that may be obtained using an opticalsensing device 111 (e.g., cameras 207). The marker device (e.g.,reference arc 115) detected by the optical sensing device 111 may be ina known fixed relationship with the surgically-relevant portion of thepatient's anatomy. The motion tracking system 105 may determine thetransformation between the marker device 115 and the optical sensingdevice 111 (e.g., using well-known triangulation techniques), and maythereby determine the transformation between the sensing device 111(e.g., camera 207 position) and the surgically-relevant portion of thepatient's anatomy. The motion tracking system 105 may similarlydetermine transformations between each of the other marker devices(e.g., marker devices 119 and 202 in FIG. 1 ) and the optical sensingdevice 111. Each of the markers 115, 119 and 202 being tracked may thenbe placed within a common coordinate system. In embodiments, the commoncoordinate system may have an origin or zero point that may beconsidered to be fixed relative to the surgically-relevant portion ofthe patient's anatomy, and may also be referred to the patientcoordinate system.

In block 305 of method 300, the first image dataset may be registered tothe common coordinate system as the second image dataset (e.g., thepatient coordinate system). This may include performing a rigidtransformation to map each pixel or voxel of the first image datasetinto corresponding 3D coordinates (i.e., x, y, z coordinates) of thecommon coordinate system. A number of techniques may be utilized forregistering multiple image datasets. In one non-limiting example of aregistration process for x-ray CT imaging data, a pre-scan calibrationprocess may be used to precisely calculate (e.g., within 1 mm) thetransformation between the isocenter of the x-ray gantry 40 and theoptical sensing device 111. A set of markers 211 (e.g., 3 or more, suchas 4-6 markers) may be provided on the surface of the gantry 40, asshown in FIG. 1 . The markers 211 may be within the field of view of theoptical sensing device 111 to enable the gantry 40 position to betracked by the motion tracking system 105. A calibration phantom (notshown for clarity) having a marker device (e.g., similar to markerdevice 115 in FIG. 1 ) fixed thereto may be placed on the patientsupport 60 such that the marker device is also within the field of viewof the optical sensing device 111. The motion tracking system 105 maydetermine the transformation between the gantry 40 coordinate systemdefined by the markers 211 and the optical sensing device 111 coordinatesystem as well as the transformation between the phantom coordinatesystem defined by the marker device on the phantom and the opticalsensing device 111 coordinate system. These transformations may be usedto determine the gantry-to-phantom transformation. The phantom may thenbe scanned using the imaging device 103. A set of elements (e.g., x-rayvisible beads) that may be easily identified from the imaging data maybe located in the phantom, where the geometry of these elements withinthe phantom coordinate system may be previously-known. An algorithm maybe used to analyze the x-ray image data to identify the x-ray visibleelements with respect to the center point of the image data, whichcorresponds to the isocenter of the gantry 40. Thus, the x-ray visibleelements may be located in a coordinate system having an origin at theisocenter of the x-ray gantry 40, and the transformations between theisocenter and the phantom and the isocenter and the markers 211 on thegantry 40 may be calculated.

During a subsequent scan of the patient 200, the position andorientation of the patient 200 with respect to the isocenter of theimaging device 103 may be determined (i.e., by tracking the positions ofthe markers 211 on the gantry 40, which are known with respect to theisocenter, and the patient reference arc 115, which is known withrespect to the surgically-relevant portion of the patient anatomy). Thismay enable the image data obtained during the scan to be registered intothe patient coordinate system.

In an alternative embodiment, the position of the optical sensing device111 may be known relative to the imaging system 103 with sufficientaccuracy such that the image dataset of the patient's anatomy obtainedusing the imaging system 103 may be registered in the common coordinatesystem of the patient without the motion tracking system 105 needing totrack the position or orientation of the imaging system 103. Inembodiments, separate markers 211 on the gantry 40 of the imaging system103 as shown in FIG. 1 may not be required or used. In some embodiments,the position of the optical sensing device 111 (e.g., the position ofeach of the cameras 207 as shown in FIGS. 1 and 2 ) may be knownrelative to the isocenter of the gantry 40 of the imaging system 103,such as via a calibration process that may be performed at the factoryor during installation or pre-calibration of the system. The gantry 40and/or the optical sensing device 111 may include keying features (e.g.,high-precision bolt patterns) where the optical sensing device 111attaches to the gantry 40 to ensure that the position of the sensingdevice 111 on the gantry 40 remains accurately fixed. In embodimentswhere the camera(s) 207 may be movable relative to the gantry 40,high-precision encoders may precisely record and correct for any changesin camera position/orientation relative to the isocenter of the gantry40. During imaging scans, the optical sensing device 111 may track theposition and orientation of the patient 200 with respect to the cameraposition, which is in a known, fixed geometric relationship with theisocenter of the imaging device 103. The image data obtained during ascan may thus be registered into the common coordinate system of thepatient without needing to first perform a calibration scan on aphantom, as described above.

In block 307 of method 300, images of the patient's anatomy from thefirst image dataset may be displayed with an overlay of one or morefeatures derived from the second image dataset in the common coordinatesystem. The images may be displayed on a suitable display device, suchas display 121 shown in FIG. 1 . The images of the patient's anatomy mayinclude 2D slices of a three-dimensional image dataset (e.g., atomographic reconstruction) and/or a 3D volume rendering of all or aportion of the image dataset. In embodiments, images obtained usingmultiple imaging devices or imaging modalities may be fused anddisplayed in a common coordinate system. For example, the first imagedataset of the patient's internal anatomy may be an x-ray CT scan.Another image dataset of the patient's internal anatomy, such as an MRIscan, may be combined with the x-ray CT data and displayed on thedisplay 121. The MRI scan data may be registered into the commoncoordinate system using a similar registration process as describedabove. Alternately or in addition, an algorithm for matching landmarksor fiducials identifiable from both image datasets may be used to mergethe datasets for display.

The one or more features derived from the second image dataset that maybe displayed overlaying the images of the patient's anatomy may includegraphical depictions of a tool 104, an end effector 102 or anotherobject that is tracked by the motion tracking system 105. The graphicaldepiction may be based on a known geometry of the tool 104, end effector102 or another object. The graphical depiction may be a rendering of theactual size and shape of the object or may be a depiction of selectfeatures of the object, such as a location of a tip end of the objectand/or an orientation of the object. The graphical depiction may alsoindicate a trajectory defined by the object (e.g., a ray extending froma tip end of the object into the patient) and/or a target point withinthe patient's anatomy that may be defined based on the position and/ororientation of one or more objects being tracked. In variousembodiments, the tool 104 may be a pointer. The tool 104 may also be asurgical instrument, such as a needle, a cannula, dilator, a tool forgripping or cutting, an electrode, an implant, a drill bit, a screw, ascrew driver, a radiation source, a drug and an endoscope. Inembodiments, the end effector 102 of the robotic arm 101 may include ahollow tube or cannula that may be configured to hold one or more tools,such as a surgical instrument, and may be used to guide an instrument asit is inserted into the patient's body. Alternately, the end effector102 itself may be or may include an instrument that may be inserted intothe patient's body.

The motion tracking system 105 may repeatedly acquire new images fromthe optical sensing device 111, and the relative positions and/ororientations of objects within the field of view of the optical sensingdevice 111 may be updated with each acquisition of new images from theoptical sensing device 111. The display 121 may be updated to reflectany change(s) in the position and/or orientation of the objects withinthe common coordinate system (e.g., relative to the patient referencearc 115), which may include adding additional graphical elements todepict new objects that are moved within the field of view of theoptical sensing device 111 and removing graphical depictions of objectswhen they are no longer within the field of view of the optical sensingdevice 111. In some embodiments, the optical sensing device 111 mayinclude a motorized system to enable the position and/or orientation ofthe camera(s) 207 to move to maintain the surgical area within thecenter of the field of view of the camera(s) 207.

FIG. 4 is a component block diagram of an image-guided surgery system400 according to an embodiment. The system 400 may be implemented usingone or more computing devices, such as computer 113 shown in FIG. 1 .The system 400 may be operatively coupled to a first display device 121,which may include a monitor that is fixed to a cart 120 or otherstructure (e.g., wall, ceiling, floor, imaging device, etc.) within theoperating suite. The system 400 may also be operatively coupled to atleast one additional display device 401, which may be a handheldcomputing device, as described above with reference to FIG. 2 . Thesystem 400 may also include an audio input/output component 403, whichmay include a speaker or other output component for outputting audiblesignals (e.g., audio instructions, alerts, etc.) and/or a microphone orother input component for receiving audio inputs (e.g., voice commands)that may be interpreted by the system 400. The system 400 may beimplemented at least partially in software and may be based on one ormore of the Image-Guided Surgery Toolkit (IGSTK), Visualization Toolkit(VTK) and Insight Segmentation and Registration Toolkit (ITK)development frameworks.

The system 400 may be configured to receive and store imaging data 407(e.g., DICOM data) collected by an imaging device 103. The imaging data407 may be received directly from the imaging device 103 or may beretrieved from another source, such as a remote server. The imaging data407 may be imaging data that is obtained prior to a surgical procedure(e.g., pre-operative image data) and/or imaging data that is obtainedduring a surgical procedure (e.g., intra-operative image data). Inembodiments, the system 400 may be configured to display themost-current image data 407 collected by the imaging device 103. Theimage data 407 may be registered to a common coordinate system as thetracking data 409 from the motion tracking system 105 in accordance witha registration method such as method 300 described above with referenceto FIG. 3 .

The system 400 may also receive tracking data 409 from a motion trackingsystem 105. The system 400 may be configured to repeatedly read thetracking data from the motion tracking system 105 indicating the currentposition/orientation of the patient and any other objects tracked by themotion tracking system 105. The system 400 may read the tracking data ata frequency (e.g., refresh rate) of greater than 100 Hz (e.g., 240 Hz).In embodiments, the tracking data from the motion tracking system 105may include data to enable the system 400 to identify particular objectsfrom within the tracking data. For example, each marker device (e.g.,marker devices 115, 202 and 119 in FIG. 1 ) may include a uniquecharacteristic (e.g., a unique geometric pattern of reflective markers,a unique flash pattern of active markers, etc.) to enable the markerdevice to be identified. These unique characteristics of the markerdevices may be registered with particular objects or tools (e.g.,associated with a particular object or tool in a database) by the system400. The unique characteristics of the marker devices may bepre-registered in the system 400 and/or may be registered to particularobjects or tools during the course of a surgical procedure.

In one embodiment, the image guided surgery system 400 may include anautomatic identification and data capture (AIDC) component 411 that maybe used during registration of surgical tools or instruments with uniquemarker devices. The AIDC component 411 may include a sensor device, suchas an optical scanner, an RF receiver, a camera, etc. that may beconfigured to analyze a characteristic of the surgical tool (e.g., scanan identifying mark, such as a model or serial number, etched into thetool, scan a barcode, RFID tag or near-field communication (NFC) tag onthe tool, analyze a geometric feature of the tool using machine vision,etc.) while the motion tracking system 105 identifies the marker deviceattached to the tool. The AIDC component may search a database todetermine whether the surgical tool or instrument has been previouslyentered into the IGS system 400, and if so, the IGS system 400 mayautomatically register the marker pattern in association with the knownsurgical tool or instrument. This may improve workflow and patientsafety by obviating the need for medical personnel to manually enterdata to pre-register tools/instruments. In embodiments, the registrationprocess for surgical tools may occur while the tool is placed within acalibration fixture that may be used to precisely determine one or moregeometric characteristics of the tool, such as the location of the tipend of the tool relative to the marker device, that may be registered inassociation with the tool and the unique marker pattern during asurgical procedure.

The system 400 may also include a library of graphical elements that maybe associated with particular objects or tools (e.g., in a database).The system 400 may display graphical elements associated with theobjects or tools being tracked by the motion tracking system 105 in thecommon coordinate system with the image data on the display(s) 119, 401.

The system 400 may include a user-interface component that may controlthe display of system information and/or graphical user interfaceelements on the display(s) 119 and 401. The system 400 may furtherprocess and implement user commands received from user interfacedevices. A user interface device, may include, for example, atouchscreen user interface which may be integrated with a display device119, 401. In embodiments, a user interface device may alternately oradditionally include one or more of a button, a keyboard, a joystick, amouse, a touchpad, etc. which may be located on a display device 119,401 and/or on a workstation (e.g., a workstation located on a cart 120).In embodiments, the user interface device(s) may also include amicrophone (e.g., audio input/output component 403) that may receivevoice commands that may be interpreted by the system (e.g., using voicerecognition software). The user commands received via one or more userinput devices may enable a user to control various functions of thesystem 400, such as changing what is shown on the display(s) 119, 401(e.g., displaying different image datasets, displaying differentslice(s) and/or different 3D rendering(s) within an image dataset,zooming in or out of an image, displaying different menu options,returning to a home screen, etc.). In embodiments, the user commands mayenable a user to set one or more trajectories and/or target locationswithin the patient's anatomy. The system 400 may store the positionsand/or orientations of user-defined trajectories or target locationswithin the common coordinate system, and may display graphicalrepresentations of such trajectories or target locations on thedisplay(s) 119, 401.

The user commands received by the system 400 may also include commandsfor controlling the operation of other components, such as the imagingdevice 103, the motion tracking system 105 and/or a robotic arm 101. Forexample, for a robotically-assisted surgical procedure, the user commandmay include an instruction to move a robotic arm 101 to a particularposition and/or orientation. The instruction to move the robotic arm 101may be based on a user interaction with image data of the patient'sanatomy that is displayed on a display device 119, 401. For example, theuser may use the display device 119, 401 to define a particulartrajectory with respect to the patient's anatomy and may send aninstruction for the robotic arm 101 to move such that that the endeffector 102 of the robotic arm 101 is positioned along the definedtrajectory.

A robotic control system 405 may control the movement of one or morerobotic arms 101. The robotic control system 405 may receive sensor dataindicating the current parameters of the robotic arm 101 (e.g., robotposition, joint angles, measured axis forces, motor currents) and maysend motor control signals to drive the movement of the arm 101. Inembodiments, the motion tracking system 105 may track the position ofthe robotic arm 101 (e.g., via marker device 202 on or proximate to endeffector 102 as shown in FIG. 1 ) to determine the position of the endeffector 102 within the common coordinate system of the patient. Acontrol loop, which may be executed using the image-guided surgerysystem 400, the motion tracking system 105 and/or the robotic controlsystem 405, may continuously read the tracking data and the robotparameter data and may send instructions to the robotic control system405 to cause the robotic arm 101 to move to a desired position andorientation.

In various embodiments, display device 119 may be a primary displaydevice (e.g., a monitor) that may be connected to the image-guidedsurgery system 400 by a wired or wireless link. In one embodiment, thesystem 400 may stream video data to the display device 119 over asuitable video data interface (e.g., an HDMI interface) and may alsoexchange other signals with the display device over a separate dataconnection (e.g., a USB connection).

In various embodiments, display device 401 may be a handheld computingdevice. A handheld display device 401 may generally be smaller andlighter than the primary display device 119 (e.g., monitor), and may incertain embodiments be referred to as a secondary display device. Insome embodiments, display device 401 may be a mirror of display device119 and may display all or a portion of the same information as is shownon display device 119. Alternately, display device 401 may displaydifferent information than is shown on display device 119. In someembodiments, display device 119 may be omitted, and handheld displaydevice 401 may be the only display device operably connected to theimage-guided surgery system 400. In such a case, display device 401 maybe referred to as the primary display device. Further, although a singlehandheld display device 401 (i.e., a tablet computer) is shown in FIG. 4, it will be understood that multiple handheld display devices 401 maybe simultaneously connected to and used with the system 400.

The handheld display device 401 may be coupled to the image-guidedsurgery system 400 by a wired or wireless communication link. In oneembodiment, the handheld display device 401 may communicate with thesystem 400 over a wireless communication interface. The system 400 maystream digital video data (e.g., high-definition video) for display onthe handheld display device 401, such as over a wireless local areanetwork (WLAN) connection, including a IEEE 801.11 (e.g., WiFi)connection. The system 400 may also exchange other signals with thehandheld display device 401 (e.g., control signals from the system 400and/or user commands received at a user interface, such as atouchscreen, on the display device 401) over a wireless connection. Thesystem 400 and the display device 401 may communicate over any suitablewireless protocol or standard, such as over a IEEE 802.15x (e.g., aBLUETOOTH®) connection.

An image-guided surgical system 400 according to various embodiments mayprovide a plurality of modes for displaying patient information. Forexample, a first display mode may include displaying a 3D image dataset(e.g., an x-ray CT, MRI, sonogram, PET or SPECT image dataset) inmultiple two-dimensional slices corresponding to anatomic planes (e.g.,axial, sagittal, coronal planes) transecting the patient. This isillustrated in the screenshot of a display device shown in FIG. 5 . Thedisplay device may be a display device 119 (e.g., monitor) as shown inFIG. 1 or a handheld display device as shown in FIGS. 2 and 4 . Thedisplay screen 500 in this example illustrates four different patientimages in four quadrants of the display screen 500. Three of thequadrants (i.e., top left, top right and bottom left quadrants ofdisplay screen 500) depict different two-dimensional slices 501, 503,505 of CT image data. A fourth quadrant (i.e., lower left quadrant ofdisplay screen 500) includes a 3D volume rendering 507 illustrating a“virtual” view of anatomic feature(s) (e.g., bony structures or otherdiscrete internal anatomic features). The two-dimensional slices 501,503, 505 correspond, respectively, to views taken along axial, sagittaland coronal planes through the patient 200.

The display screen 500 may also display graphical elements illustratingthe relationship of each slice 501, 503, 505 relative to the otherslices shown on the display screen 500. For example, as shown in FIG. 5, the axial slice 501 image data may include an overlay of a crosspattern 515 showing the intersection of the axial slice 501 with theplanes corresponding to the sagittal and coronal slices 503 and 505shown on the display screen 500. Similar cross patterns 515 may bedisplayed overlaying the display of image data in the sagittal andcoronal slices 503 and 505. The display screen 500 may also includegraphical representations or renderings of other objects or toolstracked by the motion tracking system 105. In the example of FIG. 5 , agraphical representation of a tool 509 is shown in the lower rightquadrant of the display screen 500. The graphical representation of thetool 509 may illustrate the position and orientation of the toolrelative to the anatomic features depicted in the 3D volume rendering507. Similar graphical elements may be displayed in the 2D slice images501, 503 and 505 to illustrate the position and/or orientation of one ormore objects with respect to the patient.

It will be understood that the four-quadrant view shown in FIG. 5 is onepossible implementation of a display of patient information on a displaydevice 119, 401. Other possible display modes are possible. For example,rather than illustrating multiple different images (e.g., slices) from apatient image dataset (e.g., reconstructed volume), the display screen500 may show only a single image (e.g., a single axial, sagittal orcoronal slice 501, 503, 505 or a single 3D volume rendering 507). Thedisplay screen 500 may illustrate only two slices corresponding todifferent anatomic planes (e.g., axial and sagittal, axial and coronal,or sagittal and coronal slices), or may illustrate a single slice alongwith a 3D volume rendering. In some embodiments, the display screen 500may illustrate multiple two-dimensional slices corresponding to the sameanatomic planes (e.g., multiple axial, sagittal and/or coronal slicestaken through different sections of the reconstructed volume) and/ormultiple 3D volume renderings viewed from different angles. The displayscreen 500 may also display real-time video images of the surgical area.The real-time video images may be obtained from a camera located at asuitable location, such as a head-mounted camera worn by a surgeonand/or a camera mounted to a structure (e.g., on the optical sensingdevice 111, arm 209 or support element 215, on the imaging device 103,to the patient support 60, to a surgical light apparatus, or to any ofto any of the wall, ceiling or floor in the operating room, or to aseparate cart). The video images may also be images obtained from withinthe surgical site, such as from an endoscope inserted into the patient.

The different images and display modes of the display screen 500 may becustomizable based on user selections, which may be made via a userinput device and/or user voice commands. In embodiments, the user may beable to select (e.g., scroll through) different patient images, such assequentially illustrating multiple axial, sagittal and/or coronal slicestaken through different sections of the reconstructed volume, orsequentially illustrating multiple 3D volume renderings viewed fromdifferent angles. The display screen 500 may also display slices alongoblique planes taken through the reconstructed volume. The user may alsohave the capability to control the magnification of images, such as byzooming into or out from a particular portion of an image shown in thedisplay screen 500. The user may control the selection of patient imagesfor display using a user input device, voice commands and/or via aseparate tool, such as a pointer device. In some embodiments, theintersection of the three image planes (i.e., axial, sagittal andcoronal) shown on the display panel 500 may coincide with a targetposition within the patient's body. The surgeon may use the displaypanel 500 as a “virtual cutting tool” to move through the variousslices/views of the patient image volume and to identify and select atarget region for a surgical intervention.

The user (e.g., a surgeon) may be able to set one or more targetpositions and/or trajectories within the patient 200. There may be avariety of ways to set a trajectory or target location. For example, thesurgeon may move through different views of the patient image data bymanipulating a tool (e.g., a pointer/stylus device and/or an endeffector of a robotic arm) over the patient 200, where the tool maydefine a unique trajectory into the patient. The tool may be trackedwithin the patient coordinate system using the motion tracking system105. In some embodiments, an imaginary ray projected forward from thetip end of the tool may define the unique trajectory into the patient,which may be graphically depicted on the display screen 500. A targetlocation along the unique trajectory may be defined based on apre-determined offset distance from the tip end of the tool.Alternately, the surgeon may directly manipulate and interact with thedisplayed image data to identify a particular target or trajectory, suchas using a workstation computer. A particular target point or trajectorymay be set by the system 400 in response to an input event, which mayinclude, for example, a voice command, a touch event on a touchscreeninterface, and/or an input on a user interface device (e.g., a keyboardentry, a mouse click, a button push, etc.). In embodiments, the surgeonmay set a target position and/or trajectory by interacting with imagedata displayed on a display device, such as display devices 119 and/or401. For example, the surgeon may define a target point and/ortrajectory in the patient 200 by selecting one or more points on adisplay screen 500 of a display device 119, 401 (e.g., marking thepoints using a stylus, a cursor or mouse pointer, or a touch on atouchscreen user interface). To define a trajectory, for instance, theuser may select two or more points in the image data (e.g., a targetpoint and an entrance point on the skin of the patient). In embodiments,the user may be able to make fine adjustments to a selected target pointand/or trajectory using any suitable user interface device. Multipletarget points and/or trajectories may be set and saved in a memory(e.g., in an image-guided surgery system 400 as illustrated in FIG. 4 ),where each target point and/or trajectory may be saved in associationwith a unique identifier (e.g., file name).

In embodiments, the display screen 500 may display graphical element(s)overlaying the image data corresponding to one or more target locationsand/or trajectories that are set by the user. For example, definedtarget locations may be illustrated as identifiable dots or points inthe image data, which may be color coded and/or labeled on the displayscreen 500 to enable easy visualization. Alternately or in addition,defined trajectories may be depicted as identifiable lines or linesegments in the image data, which may be similarly color coded and/orlabeled. As discussed above, the display screen 500 may also displaygraphical elements associated with particular tools or objects,including invasive surgical tools or instruments that are tracked by themotion tracking system 105. In embodiments, the display screen 500 maydepict at least a portion (e.g., a tip end) of a surgical instrument asit is inserted into the patient 200, which may enable the surgeon totrack the progress of the instrument as it progresses along a definedtrajectory and/or towards a defined target location in the patient 200.In some embodiments, the patient images on the display screen 500 may beaugmented by graphical illustrations of pre-calibrated tools or implants(e.g., screws) that are located within the patient 200.

The at least one robotic arm 101 may aid in the performance of asurgical procedure, such as a minimally-invasive spinal surgicalprocedure or various other types of orthopedic, neurological,cardiothoracic and general surgical procedures. In some embodiments,when the robotic arm 101 is pointed along a set trajectory to a targetposition, the robotic arm 101 may maintain a rigid or fixed pose toenable the surgeon to insert an instrument or tool through a cannula orsimilar guide arranged along a vector that coincides with the predefinedtrajectory into the body of the patient 200. The cannula may be aportion of the end effector 102 of the robotic arm 101 or it may beseparate component that is held by the end effector 102. Thecannula/guide may be positioned by the robotic arm 101 such that thecentral axis of the cannula is collinear with the pre-defined trajectoryinto the patient 200. The surgeon may insert one or more invasivesurgical instrument through the cannula/guide along the trajectory andinto the body of the patient to perform a surgical intervention.Alternately, the end effector 102 itself may comprise a surgicalinstrument that may be moved into the body of the patient, such as,without limitation, a needle, a dilator, a tool for gripping, cutting orablating tissue, an implant, a drill bit, a screw, a screw driver, aradiation source, a drug and/or an endoscope.

Various embodiments include an image guided surgery system that has anoptical system that provides a visible indication of a range of a motiontracking system. In various embodiments, the optical system includes atleast one light source that directs visible light to indicate afield-of-view of one or more optical sensing devices (e.g., cameras) ofa motion tracking system. FIGS. 6A and 6B illustrate an example of anoptical sensing device 111 of a motion tracking system 105, such asdescribed above with reference to FIGS. 1 and 2 . The optical sensingdevice 111 includes an array of multiple (e.g., four) cameras 107mounted to a rigid support 208 (see FIG. 6B). The support 208 may bemounted above the surgical area, such as on an arm 209 as shown in FIG.2 . The support 208 in this embodiment includes a handle 210 thatextends from the bottom of the support 208 that may be used to adjustthe position and/or orientation of the optical sensing device 111.

As discussed above, an optically-based motion tracking system 105 mayinclude a stereoscopic camera array that detects optical radiation(typically infrared (IR) radiation) from a plurality of marker devices.The markers may be active markers that include IR emitters or may bepassive markers that reflect IR radiation from an external source, whichmay be co-located with the camera array. In either type of motiontracking system 105, it may be difficult for the user to determine whichobjects are within the field-of-view of the camera array at a given time(and thus are being tracked) and whether the cameras' line of sight tothe surgical area is blocked by an obstruction.

Embodiments include an optical system for providing a visible indicationof the range of a motion tracking system 105, including a field-of-viewof an optically-based motion tracking system. As shown schematically inFIGS. 6A-6B, an optical sensing device 111 includes a visible lightsource 601, which may be a laser source, mounted to the rigid support208 that supports an array of cameras 107. Although a single visiblelight source 601 is shown in FIG. 6A, it will be understood thatmultiple visible light sources may be mounted to the support 208. Inthis embodiment, the visible light source 601 is mounted inside thehandle 210 of the support 208. The visible light source 208 projects abeam 603 of radiation in a first (e.g., vertical) direction onto thesurface of a rotating mirror 605, as shown in the partial cross-sectionview of FIG. 6B. The rotating mirror 605 redirects the beam 603 along asecond (e.g., horizontal) direction that is substantially perpendicularto the first direction. A motor 607 mechanically coupled to the rotatingmirror 605 rotates the mirror within the rigid support 208. The rotationof the mirror 605 causes the beam 603 to revolve around the support. Asecond set of mirrors 609 spaced radially from the rotating mirror 605redirects the beam 603 along a third direction towards the surgicalsite. As shown in FIG. 6A, the second set of mirrors 609 may be adjacentto and optionally coupled to the cameras 107 of the camera array. Eachof the mirrors 609 may have an angled or contoured surface thatredirects the beam in the direction of the camera's 107 field of view.As the rotating mirror 605 rotates around the support 608, the beam 603repeatedly traces a visible outline 613 on whatever surface(s) arelocated in front of the camera array. At a sufficiently high rate ofrevolution of the rotating mirror 605, the outline 613 may becontinuously perceptible to the user. The second set of mirrors 609 maybe calibrated so that the visible outline 613 generally corresponds to aboundary of a field-of-view of the camera array. For example, thevisible outline 613 may be a circle or arc that encompasses an outerboundary of the camera's 607 field of view, as shown in FIGS. 6A-6B. Inone embodiment shown schematically in FIG. 6C, the visible outline 613traced by the beam may encompass a region 615 that is within the fieldof view of at least two cameras 107 of the camera array.

The visible light source 601 may also be a source of incoherent light,such as a light emitting diode (LED), as an alternative to a laser asdescribed above. An advantage of a laser source is that may be used tocreate a sharply-delineated boundary. However, a drawback to the use ofa laser light beam is that it may reflect off of shiny surfaces,including instruments, and can create safety issues. The visible lightsource 601 may be a high-intensity non-laser light source, such as anLED, which may be configured to reflect off of a 360-degree reflector toproduce a disc of light. The 360-degree reflector may be an alternativeto the rotating mirror 605 as described above. The disc of light fromthe reflector may be directed to reflect off the aforementioned angledor contoured mirrors 609 to project a pattern of illumination whichoverlaps the field-of-view of the cameras 107.

In embodiments, the optical sensing device 111 may be positioned tooptimize the view of the cameras 107 into the surgical space. However,it may be desirable for the cameras 107 to also see marker devices thatare located outside of the surgeon's work space, such one or moremarkers attached to the base end of the robotic arm 101 (e.g., toprovide a “ground truth” measurement of robot position) and/or on theimaging device 103, such as the markers 211 on the gantry 40 used forscan registration as shown in FIG. 1 . The cameras 107 may have a largeenough field of view to see all of the markers if the cameras 107 arepositioned far from the patient, but such a camera position may besub-optimal for viewing and tracking instruments within the surgicalarea.

FIG. 7A illustrates an embodiment of an image guided surgery system thatincludes a motion tracking system 105 having an optical sensing device111 as described above. The optical sensing device 111 includes an arrayof cameras 107 mounted to a rigid support 208. The support 208 may bemounted above the surgical area, with the cameras 107 pointed in a firstdirection to detect optical signals from one or more first markerarray(s) 703 a, 703 b located in the surgical area. In the embodiment ofFIG. 7A, marker arrays 703 a, 703 b are shown attached to the patient200 and the robotic arm 101, respectively. The first marker array(s) mayinclude active markers that emit their own optical signals (e.g., IRradiation), or passive markers that reflect optical signals from anexternal source. At least one beam splitter 701 is optically coupled tothe optical sensing device 111. In the embodiment shown in FIG. 7A, eachcamera 107 of the optical sensing device 111 includes an associated beamsplitter 701 positioned in front of the camera 107. The beam splitters701 enable optical signals from the first marker array(s) 703 a, 703 bin the surgical area to pass through the beam splitters 701 along afirst direction to be detected by the cameras 107. The beam splitters701 are configured redirect optical signals received along a seconddirection from one or more second marker array(s) 705 located outside ofthe surgical area so that the signals may be received by the cameras107. The one or more second marker array(s) 705 may be located on orproximate to the base end of the robotic arm 101 or on the imagingdevice 103, for example. The one or more second marker array(s) 705 maybe referred to as reference markers that are located outside of theactive surgical site.

In some embodiments, both the first marker array(s) 703 a, 703 b in thesurgical area and the second marker array(s) 705 located outside of thesurgical area may be active marker arrays. The operation of the markerarrays 703, 705 may be synchronized so that the cameras 107 receivesignals from the first marker array(s) 703 a, 703 b and the secondmarker array(s) 705 at different times.

Alternately, the first marker arrays 703 a, 703 b and the second markerarrays 705 may be passive maker arrays that reflect IR radiation. Themotion tracking system 105 may be configured to capture tracking datafrom the first direction and from the second direction at differenttimes by, for example, projecting IR radiation along the first directionand along the second direction at different times.

In some embodiments, the motion tracking system 105 may be a hybridsystem that utilizes both active and passive markers. In one example,the first marker array(s) 703 a, 703 b in the surgical area may bepassive makers and the second marker array(s) 705 outside of thesurgical area (e.g., at the base of the robotic arm 101 and/or theimaging system 103) may be active makers. The operation of the activesecond marker array(s) 705 may be synchronized with an IR source thatprojects IR radiation into the surgical area so that when the cameras107 are receiving reflected radiation from the first marker array(s) 703a, 703 b the second marker array(s) 705 are not emitting, and when thecameras 107 are receiving radiation emitted by the second markerarray(s) 705 the IR source is not projecting into the surgical area.

As noted above, the optical sensing device 111 of the motion trackingsystem 105 may include a plurality of cameras 107 mounted to rigidsupport 208. The rigid support 208 may maintain the cameras 107 in afixed relationship relative to one another. However, depending on howthe rigid support 208 is mounted within the operating room, there canoccur small movements (e.g., vibration, shaking, etc.) of the rigidsupport 208 and cameras 107 relative to the patient 200 and/or roboticarm 101. The optical sensing device 111 may also be repositioned by theuser. The software of the motion tracking system 105 may not be able todistinguish between movements that are actual movements of the objectsbeing tracked (such as marker arrays 703 a, 703 b within the surgicalarea) and an apparent movement of the tracked object(s) due to motion ofthe cameras 107 themselves. This may result in decreased accuracy of thesurgical navigation and unnecessary movements of the robotic arm 101 tocompensate for apparent motions of objects (such as the patient 200and/or robotic arm 101) within the surgical field.

FIG. 7B illustrates an embodiment of an image guided surgery system thatincludes a motion tracking system 105 having an inertial measurementunit 711 attached to an optical sensing device 111. The inertialmeasurement unit 711 may be mounted to the rigid support 208 on which anarray of cameras 107 are mounted. The inertial measurement unit 711 maydetect movements of the optical sensing device 111, such as a shaking orvibration of the rigid support 208 holding the cameras 107, or anintentional or accidental repositioning of the rigid support by a user.The measurements detected by the inertial measurement unit 711 may besent to a processing device (e.g., a computer 113, such as shown in FIG.1 ) along with tracking data from the optical sensing device 111. Themeasurements from the inertial measurement unit 711 may be utilized bymotion tracking software implemented on the computer 113 to correct fordetected movements of the optical sensing device 111 when trackingobjects within the surgical area, including the patient 200, the roboticarm 101 and surgical tools. The inertial measurement unit 711 mayprovide an measurement of movement of the optical sensing device 111that is independent of the tracking data obtained by the optical sensingdevice 111, which can aid the motion tracking system 105 in accuratelydifferentiating between actual movements of the objects being trackedfrom apparent movements of the objects due to motion of the opticalsensing device 111 itself.

The inertial measurement unit 711 may include a three-axis accelerometerand a three-axis gyroscope. The accelerometer and gyroscope may befabricated utilizing MEMS technology. The accelerometer and gyroscopemay be separate components (e.g., chips) located in the rigid support208 or may be integrated on a single device (e.g., integrated circuit).The inertial measurement unit 711 may also include circuitry coupled tothe accelerometer and gyroscope that may be configured to read outputsignals from these components. The accelerometer may output signalsmeasuring the linear acceleration of the rigid support 208, preferablyin three-dimensional space. The gyroscope may output signals measuringthe angular velocity of the rigid support, preferably also inthree-dimensional space. The signals from the accelerometer andgyroscope may be processed using a suitable processor, such as acomputer 113 shown in FIG. 1 , to determine the position and orientationof the rigid support 208 with respect to an initial inertial referenceframe via a dead reckoning technique. In particular, integrating theangular velocity measurements from the gyroscope may enable the currentorientation of the rigid support 208 to be determined with respect to aknown starting orientation. Integrating the linear accelerationmeasurements from the accelerometer may enable the current velocity ofthe rigid support 208 to be determined with respect to a known startingvelocity. A further integration may enable the current position of therigid support 208 to be determined with respect to a known startingposition.

In embodiments, measurement data from the inertial measurement unit 711may transmitted from the optical sensing device 111 to a separatecomputing device (e.g., computer 113) via a wired or wireless link. Themeasurement data from the inertial measurement unit 711 may be sent viathe same communication link as the tracking data from the cameras 107,or by a different communication link.

Although the embodiment of FIG. 7B illustrates an inertial measurementunit 711 located on the rigid support 208 holding the cameras 107, itwill be understood that the inertial measurement unit 711 may be locatedon a camera 107. Each camera 107 may include an inertial measurementunit 711 that measures the motion of the camera 107 to correct forcamera movement in the motion tracking system 105.

FIG. 8A illustrates an embodiment of a robotic arm 101 for use inrobotic-assisted surgery. The robotic arm 101 includes a base end 801and a distal end 803, with an end effector 102 located at the distal end803. As discussed above, the end effector 102 may comprise a cannula orguide that may be used to insert an invasive surgical tool or instrumentalong a trajectory into the patient, or alternately, the end effector102 itself may comprise an invasive surgical instrument. During asurgical procedure, the robotic arm 101 may be covered in a surgicaldrape to maintain a sterile surgical field. The end effector 102 may bea sterile component that may be attached (e.g., snapped into) the distalend 803 of the robotic arm 101, optionally over the surgical drape.

The robotic arm 101 in this embodiment includes a first portion 805 thatincludes at least one 2-DOF joint 806. As used herein a “2-DOF joint” isa joint that enables robot articulation in two mutually orthogonaldirections (i.e., pitch and yaw rotation). A 2-DOF joint is in contrastto a conventional (i.e., 1-DOF) robotic joint that rotates within asingle plane. In the embodiment of FIG. 8A, the first portion 805 of therobotic arm 101 includes a chain of five 2-DOF joints 806 extending fromthe base end 801 of the robotic arm 101, although it will be understoodthat the first portion 805 may have more or less 2-DOF joints 806. The2-DOF joints 806 may be modular in design, with each joint 806 includinga central section 808 having a generally spherical outer surface betweena pair of end sections 810 a, 810 b. The end sections 810 a, 810 b ofadjacent joints 806 may be connected. The central section 808 mayinclude a pair of wedge segments 812 a, 812 b having angled interfacingsurfaces so that the rotation of the wedge segments 812 a, 812 brelative to one another produces pitch and yaw rotation of end section810 b relative to end section 810 a over a particular angular range(e.g., ±15 degrees, such as ±20 degrees, ±30 degrees, ±45 degrees and±100 degrees or more). Motors (not illustrated) mounted to the endsections 810 a, 810 b may drive the rotation of the wedge segments 812a, 812 b. A universal joint (not visible) located inside the centralsection 808 and coupled to the end sections 810 a, 810 b may inhibittwisting motion of the end sections 810 a, 810 b.

The robotic arm 101 in FIG. 8A may also include a second portion 807that includes at least two rotary joints 814 a, 814 b (i.e., 1-DOFjoints) that rotate about mutually perpendicular axes. The secondportion 807 may include a housing 815 that includes motors (not visible)for driving the rotation of joints 814 a and 814 b. The second portion807 may be located between the first portion 805 and the end effector102 of the robotic arm 101. Joint 814 a may be a theta wrist joint thatmay rotate the entire housing 815 with respect to the end section 810 bof the adjacent 2-DOF joint 806. In embodiments, joint 814 a may rotatecontinuously (e.g., >360°) in either direction. Joint 814 b may be awrist joint located on the side of the housing 815. Joint 814 b mayrotate the end effector 102 at least about ±45 degrees, such as ±90degrees, ±120 degrees, ±170 degrees or more with respect to the housing815. In one embodiment illustrated most clearly in FIG. 8B, a connector816 between the joint 814 b and the end effector 102 may align the endeffector 102 with the midline of the housing 815 so that the rotation ofthe end effector 102 via joint 814 b is in the same plane as the axis ofrotation of theta wrist joint 814 a. Alternately, the end effector 102may rotate in a plane that is off-set from the axis of joint 814 a, asshown in the embodiment of FIGS. 8D-8E.

Further embodiments include marker arrays for tracking the positionand/or orientation of an end effector 102 of a robotic arm 101 using amotion tracking system 105. It may be desirable to provide a markerarray that is proximate to the end effector 102 of the robotic arm 101to maximize the accuracy of the positioning of the end effector 102.Conventional marker arrays include rigid frames having marker elementsaffixed thereon. Such arrays may project into the surgeon's workspaceand may interfere with a surgical procedure.

In the embodiment shown in FIG. 8C, a marker array 820 includes aplurality of marker elements attached to the robotic arm 101, and inparticular, a plurality of marker elements attached to the secondportion 807 of the robotic arm 101 proximate to the end effector 102.The marker array 820 includes a first set of one or more markers 821located distal to the most distal joint 814 b of the robotic arm 101 anda second set of one or more markers 823 located proximal to the mostdistal joint 814 b of the robotic arm 101. Together, the first andsecond sets of markers 821, 823 form a marker array 820 that may betracked by a motion tracking system 105 as described above.

In the embodiment of FIG. 8C, the second set of markers includes aplurality of markers 823 that may be secured to the housing 815 of thesecond portion 807 of the robotic arm 101. The second set of markers 825may be spaced circumferentially around the housing 815. The first set ofmarkers includes a single marker 821 that is located distal to the mostdistal joint of the robotic arm (i.e., joint 814 b). Marker 821 may belocated on the connector 816 that connects joint 814 b to the endeffector 102. Alternately, marker 821 may be located on the end effector102 itself. Although the embodiment of FIG. 8C illustrates the first setof markers as consisting of a single marker 821, it will be understoodthat a plurality of markers 821 may be located distal to joint 814 b.

The second set of markers 823 may be disposed in a geometric patternthat may be detected by the motion tracking system 105 and used todetermine both the position of the robotic arm 101 in three-dimensionalspace as well as the rotational position of theta wrist joint 814 a. Asthe end effector 102 is rotated on joint 814 b, the change in relativeposition of the first set of marker(s) (i.e., marker 821 in FIG. 8C) tothe second set of markers 823 detected by the motion tracking system 105may be used to accurately determine the position and orientation of theend effector 102. The markers 821, 823 may be relatively unobtrusive soas not to interfere with the patient or the surgeon's work space.

In some embodiments, the markers 821, 823 attached to the robotic arm101 may be active (i.e., light emitting) markers. The electrical powerfor light-emitting elements of the markers 821, 823 may be providedthrough the robotic arm 101. Alternately, the markers 821, 823 may bepassive markers (e.g., spherical elements having a retroflectivecoating) that reflects light from an external source.

FIGS. 9A and 9B illustrate embodiments of an active marker device 901that is located on a robotic arm 101. Power for the marker device 901may be provided via a conductor 902 through the robotic arm 101 to alight source 903 (e.g., an LED source that emits light in an infrared(IR) wavelength range). In the embodiment of FIG. 9A, a projection 904comprising an optical guide 905 (e.g., a light pipe) projects from thesurface 906 of the robotic arm 101. Light from the light source 903 iscoupled into the optical guide 905 and is transmitted through theprojection 904. An optical diffuser 907 is attached to the projection904 over the optical guide 905 and scatters the light as it emerges fromthe guide 905. In the embodiment shown in FIG. 9A, the robotic arm 101is covered by a surgical drape 908. The surgical drape 908 is at leastpartially light-transmissive. The diffuser 907 may be attached to theprojection 904 over the surgical drape 908. Mating features 909 on theprojection 904 and the diffuser 907 may enable the diffuser 907 to snapon to the projection 904 over the surgical drape 908, which may be heldtight against the projection 904. The diffuser 907 may be a sterilecomponent, and may be a single-use disposable component.

FIG. 9B illustrates an alternative embodiment in which the robotic arm101 includes a recessed portion 910, and the optical diffuser 907includes a projection 911 that may be inserted into the recessed portion910 to secure the diffuser 907 to the robotic arm 101. The diffuser 907may be inserted over a surgical drape 908. Light from the light source903 may be directed through the bottom of the recessed portion 910 andcoupled into an optical guide 912 in the projection 911 of the opticaldiffuser 907. The projection 911 and the recessed portion 910 mayoptionally have mating features such as shown in FIG. 9A that secure theoptical diffuser 907 to the robotic arm 101. As in the embodiment ofFIG. 9A, the diffuser 907 may be a sterile component, and may be asingle-use disposable component.

FIGS. 10A-10E illustrate various embodiments of an active marker array1001 that may be used to track various objects and tools during imageguided surgery. For example, the marker device 1001 may be a referencemarker device that is attached to the patient (e.g., reference markerdevice 115 as shown in FIG. 1 ), it may be a marker device that isattached to a surgical tool or instrument (e.g., marker device 119 asshown in FIG. 1 ), and/or it may be a marker device that is attached toa robotic arm 101 for robot-assisted surgery (e.g., marker device 202 asshown in FIG. 1 ).

In general, an active marker device 1001 according to variousembodiments includes a rigid frame 1003, an electronics module 1005 thatincludes at least one light source 1006 (e.g., an LED that emits lightin an infrared range), and an optical guide apparatus 1007 that coupleslight from the at least one light source 1006 to an array of emitterlocations 1008 on the rigid frame 1003. In embodiments, the rigid frame1003 may be made (e.g., machined) to precise dimensions and tolerancesout of metal or another suitable structural material. The rigid frame1003 may include a network of channels 1009 extending within the rigidframe 1003. The optical guide apparatus 1007 may be located within thechannels 1009. The channels 1009 may terminate in openings 1011 in theframe 1003 which may define the emitter locations 1008 of the markerdevice 1001.

FIG. 10A is a side cross-sectional view of an active marker device 1001according to an embodiment. In this embodiment, the frame 1003 mayinclude a recess 1012 or housing within which the electronics module1005 may be located. In addition to the at least one light source 1006,the electronics module 1005 may also include a power source (e.g., abattery) and circuitry for controlling the operation of the at least onelight source 1006. For example, the circuitry may control the at leastone light source 1006 to emit light having a particular pulse pattern orfrequency. In embodiments, the circuitry may comprise or include aprogrammable microprocessor. The electronics module 1005 may alsoinclude communications circuitry, such as an RF receiver and/ortransmitter, to enable communication with an external device, such ascomputer 113 in FIG. 1 . In embodiments, the electronics module maycommunicate with an external device for synchronization of the pulsingof the light source and/or for setting or programming a particular pulsepattern. In embodiments, the electronics module 1005 may be locatedwithin a sealed housing or package. The power source may be arechargeable battery, and in embodiments may be recharged wirelessly(e.g., via inductive charging).

The optical guide apparatus 1007 in this embodiment comprises aplurality of light pipes 1013. The light pipes may be made of athermoplastic material (e.g., polycarbonate) and may be at leastpartially flexible or deformable. Alternately, the optical guideapparatus 1007 may comprise a plurality of optical fibers. The opticalguide apparatus 1007 may comprise a unitary component. The optical guideapparatus 1007 may be separate from the electronics module 1005 or maybe integral with the electronics module 1005. For example, theelectronics module 1005 may be formed as a flex circuit such as shown inFIG. 10C, where a plurality of light pipes 1013 or similar opticalguides may extend from the flex circuit.

FIG. 10B is an exploded view of an active marker device 1001, includinga rigid frame 1003, an electronics module 1005, and an optical guideapparatus 1007. In this example, the optical guide apparatus 1007includes a plurality of light pipes 1009 extending from a protectivecover 1031. The cover 1031 can attach to (e.g., snap into) the rigidframe 1003 to enclose the electronics module 1005 within a recess 1012in the rigid frame 1003. In some embodiments, the electronics module1005 may be attached to the protective cover 1031 so that theelectronics module 1005 and optical guide apparatus 1007 form anintegral component.

Alternatively, the electronics module 1005 may be a separate componentthat may be inserted within the recess 1012 of the rigid frame 1003, andthe optical guide apparatus 1007 may be attached over the electronicsmodule 1005. FIG. 10B also illustrates a separate power source 1033(e.g., battery) for the electronics module 1005 that may be housedwithin the rigid frame 1003. Alternately, the power source 1033 may beintegrated with the electronics module 1005.

The optical guide apparatus 1007 may be positioned within the frame 1003such that light from the at least one light source 1006 of theelectronics module 1005 is coupled into the light pipes 1013 of theoptical guide apparatus 1007. Each of the light pipes 1013 may beinserted within a respective channel 1009 of the frame 1003. The lightpipes 1013 may terminate proximate to the respective openings 1011 ofthe frame 1003 corresponding to the emitter locations 1008. Inembodiments, optical diffusers 1017, which may be similar to thediffusers 907 described above in connection with FIGS. 9A-9B, may belocated at the ends of the light pipes 1013 and over the openings 1011in the rigid frame 1003. The diffusers 1017 may be integrated with theoptical guide apparatus 1007 or may be snap-on components that attachover the ends of the light pipes 1013 and/or the openings 1011 in therigid frame 1003.

FIG. 10C illustrates an alternative embodiment of an electronics module1005 for an active marker device 1001 that is formed as a flex circuit.The electronics module 1005 includes a flexible substrate 1040 on whichis located circuitry 1045 for controlling the operation of at least onelight source 1006. The circuitry 1045 may contain, for example, aprogrammable microprocessor, and may also include communicationscircuitry, such as an RF receiver and/or transmitter, and an integratedpower source as described above. The flexible substrate 1040 includes aplurality of peripheral arm regions 1041, each having a light source1006 (e.g., LED) attached thereto. Electrical conductors 1043 may extendalong the arm regions 1041 to connect each of the light sources 1006 tothe rest of the circuitry 1045. The light sources 1006 may be located onor proximate to the distal ends of the arm regions 1041. The electronicsmodule 1005 as shown in FIG. 10C may be inserted into a rigid frame1003, such as shown in FIGS. 10A-B. The arm regions 1041 may extendalong channels 1009 in the rigid frame 1003 so that each light source1006 is aligned with a respective opening 1011 of the frame 1003corresponding to emitter locations 1008 of the active marker device1001.

FIG. 10D illustrate an additional embodiment of an active marker device1001 in which the electronics module 1005 is housed within the handle1019 of a surgical tool 104. A rigid frame 1003 as described above maycomprise a part or all of the handle 1019. The rigid frame 1003 includesa plurality of arms 1021 that extend from the handle 1019 of the tool104. Each arm 1021 includes a channel 1009 that extends along the lengthof the arm 1021 to an opening 1011 at the end of the arm 1021. Theoptical guide apparatus 1007 includes a plurality of optical fibers 1023that are located within respective channels 1009 of the frame 1003. Theends of the optical fibers 1023 may be positioned so that light isdirected out of the respective openings 1011. The optical fibers 1023direct light from at least one light source 1006 in the handle 1019along the arms 1021 and out of the respective openings 1011. A pluralityof diffusers 1017 may be located over the openings 1011.

The electronics module 1005 may be integrated with the rigid frame 1003or may be removable from the frame 1003. FIG. 10D also illustrates apower source 1033 (e.g., battery) within the frame 1003 for providingpower to the electronics module 1005. Alternatively, the electronicsmodule 1005 may have an integrated power supply 1005. The rigid frame1003 may be detachable from the rest of the surgical tool 104, or may beintegral with the surgical tool 104.

In embodiments of an active marker device 1001 as described above, theoptical guide apparatus 1007 may optionally be removable from the rigidframe 1003 and the components may be separately sterilized for reuse ordisposed. In embodiments in which the rigid frame 1003, optical guideapparatus 1007 and electronics module 1005 are comprised of separatecomponents, each of these components may be individually removed andseparately sterilized for reuse or disposed. In some embodiments, one ormore of the rigid frame 1003, optical guide apparatus 1007 andelectronics module 1005 may be a single-use disposable component.

A plurality of active marker devices 1001 may utilize an identicaldesign for the rigid frame 1003, with the differentiation betweenmarkers provided by differences in the pulse patterns produced by theelectronics module 1005. This may provide an economical marker device1001 that may be optimized for ergonomics or other factors.

In some embodiments, the electronics component may include at least onefirst light source that emits light (e.g., IR light) that is detectableby the motion tracking system 105 for tracking purposes as describedabove, and at least one second light source that emits visible light.The visible light from the at least one second light source may becoupled into the optical guide apparatus 1007 to provide the user withvisual feedback from the marker device 1001. The visual feedback mayprovide feedback on the operation of the marker device 1001 itself(e.g., an indication of the charge state of the battery, an indicationof whether the marker device is on, programmed and actively emitting IRlight, etc.). In some embodiments, the electronics module 1005 mayreceive feedback data from an external device (such as computer 113) andmay control the at least one second light source to provide visualfeedback to the user based on the received feedback data. The visualfeedback may provide feedback regarding a surgical procedure. Forexample, the at least one second light source may flash light of acertain color (e.g., green) when the tool to which the marker device1001 is attached is determined to be in the correct position (e.g., at atarget location or along a pre-set trajectory within the patient) andmay flash a different color (e.g., yellow) when the tool is in anincorrect position. In addition, the visual feedback may indicatewhether or not a tool is currently being tracked by the motion trackingsystem 105. The at least one first (IR) light source and the at leastone second (visible) light source may be multiplexed so that only onesource is emitting at a time.

Alternatively or in addition, a cover 1031 of the active marker device1001 (see FIG. 10B) may comprise a transparent material over at least aportion of the cover 1031 to enable the user to view visual feedbackfrom the at least one second light source through the cover 1031.

Further embodiments include a motion tracking system 105 forrobotic-assisted image guided surgery that utilize an “inside out”architecture in which the sensors for tracking the marker devices arelocated on a robotic arm 101. An exemplary embodiment is illustrated inFIGS. 11A and 11B. In this embodiment, the robotic arm 101 includes asensor array 1101 extending around the circumference of the arm 101proximate to the end effector 102. The sensor array 1101 may include aplurality of outward-facing cameras, which may be similar to the cameras107 described above with reference to FIGS. 1 and 2 . The sensor array1101 may also include one or more IR light sources for illuminatingreflective marker. The sensor array 1101 may be configured to detectradiation from passive or active marker devices 1103 within the field ofview of the sensor array 1101, which may be used to track the positionsof the marker devices 1103 relative to the sensor array 1101 (e.g.,using triangulation techniques) as described above. The position of thesensor array 1101 and of the end effector 102 within a common referenceframe may be determined based on the robot joint coordinates and theknown geometry of the robotic arm 101.

As shown in FIG. 11A, the sensor array 1101 may simultaneously trackmultiple marker devices 1103, including a first marker device 1103 aattached to the patient and a second marker device 1103 b attached to atool 104 that is inserted in the end effector 102 of the robotic arm101. The field of view of the sensor array 1101 may extend around theentire circumference of the robotic arm 101, and may encompass theentire surgical area. In embodiments, a separate marker device on therobotic arm 101 may not be necessary. Further, the need for an externaloptical sensing device, such as sensing device 111 shown in FIGS. 1 and2 , may be avoided.

In some embodiments, the sensor array 1103 may also be used forcollision avoidance for the robotic arm 101. Optionally, the sensorarray 1103 may include a user interface component, such as a touchscreendisplay interface, for controlling various operations of the robotic arm101 and/or the image guided surgery system.

FIG. 11B illustrates the robotic arm 101 moved to a second positionduring an imaging scan using the imaging device 103. In this position,the sensor array 1103 has a clear view of both the patient marker device1103 a and additional markers 1104 on the imaging gantry 40, which mayfacilitate registration of image scan data, as discussed above in FIG. 3. The control system for the robotic arm 101 may be configured to movethe robotic arm 101 to optimally position the sensor array 1103 for scandata registration during an imaging scan.

FIG. 12 is a system block diagram of a computing device 1300 useful forperforming and implementing the various embodiments described above. Thecomputing device 1300 may perform the functions of an image guidedsurgery system 400 and/or a robotic control system 405, for example.While the computing device 1300 is illustrated as a laptop computer, acomputing device providing the functional capabilities of the computerdevice 1300 may be implemented as a workstation computer, an embeddedcomputer, a desktop computer, a server computer or a handheld computer(e.g., tablet, a smartphone, etc.). A typical computing device 1300 mayinclude a processor 1301 coupled to an electronic display 1304, aspeaker 1306 and a memory 1302, which may be a volatile memory as wellas a nonvolatile memory (e.g., a disk drive). When implemented as alaptop computer or desktop computer, the computing device 1300 may alsoinclude a floppy disc drive, compact disc (CD) or DVD disc drive coupledto the processor 1301. The computing device 1300 may include an antenna1310, a multimedia receiver 1312, a transceiver 1318 and/orcommunications circuitry coupled to the processor 1301 for sending andreceiving electromagnetic radiation, connecting to a wireless data link,and receiving data. Additionally, the computing device 1300 may includenetwork access ports 1324 coupled to the processor 1301 for establishingdata connections with a network (e.g., LAN coupled to a service providernetwork, etc.). A laptop computer or desktop computer 1300 typicallyalso includes a keyboard 1314 and a mouse pad 1316 for receiving userinputs.

The foregoing method descriptions are provided merely as illustrativeexamples and are not intended to require or imply that the steps of thevarious embodiments must be performed in the order presented. As will beappreciated by one of skill in the art the order of steps in theforegoing embodiments may be performed in any order. Words such as“thereafter,” “then,” “next,” etc. are not necessarily intended to limitthe order of the steps; these words may be used to guide the readerthrough the description of the methods. Further, any reference to claimelements in the singular, for example, using the articles “a,” “an” or“the” is not to be construed as limiting the element to the singular.

The various illustrative logical blocks, modules, circuits, andalgorithm steps described in connection with the embodiments disclosedherein may be implemented as electronic hardware, computer software, orcombinations of both. To clearly illustrate this interchangeability ofhardware and software, various illustrative components, blocks, modules,circuits, and steps have been described above generally in terms oftheir functionality. Whether such functionality is implemented ashardware or software depends upon the particular application and designconstraints imposed on the overall system. Skilled artisans mayimplement the described functionality in varying ways for eachparticular application, but such implementation decisions should not beinterpreted as causing a departure from the scope of the presentinvention.

The hardware used to implement the various illustrative logics, logicalblocks, modules, and circuits described in connection with the aspectsdisclosed herein may be implemented or performed with a general purposeprocessor, a digital signal processor (DSP), an application specificintegrated circuit (ASIC), a field programmable gate array (FPGA) orother programmable logic device, discrete gate or transistor logic,discrete hardware components, or any combination thereof designed toperform the functions described herein. A general-purpose processor maybe a microprocessor, but, in the alternative, the processor may be anyconventional processor, controller, microcontroller, or state machine. Aprocessor may also be implemented as a combination of computing devices,e.g., a combination of a DSP and a microprocessor, a plurality ofmicroprocessors, one or more microprocessors in conjunction with a DSPcore, or any other such configuration. Alternatively, some steps ormethods may be performed by circuitry that is specific to a givenfunction.

In one or more exemplary aspects, the functions described may beimplemented in hardware, software, firmware, or any combination thereof.If implemented in software, the functions may be stored on as one ormore instructions or code on a non-transitory computer-readable medium.The steps of a method or algorithm disclosed herein may be embodied in aprocessor-executable software module executed which may reside on anon-transitory computer-readable medium. Non-transitorycomputer-readable media includes computer storage media that facilitatestransfer of a computer program from one place to another. A storagemedia may be any available media that may be accessed by a computer. Byway of example, and not limitation, such non-transitorycomputer-readable storage media may comprise RAM, ROM, EEPROM, CD-ROM orother optical disk storage, magnetic disk storage or other magneticstorage devices, or any other medium that may be used to carry or storedesired program code in the form of instructions or data structures andthat may be accessed by a computer. Disk and disc, as used herein,includes compact disc (CD), laser disc, optical disc, digital versatiledisc (DVD), floppy disk, and blu-ray disc where disks usually reproducedata magnetically, while discs reproduce data optically with lasers.Combinations of the above should also be included within the scope ofnon-transitory computer-readable storage media. Additionally, theoperations of a method or algorithm may reside as one or any combinationor set of codes and/or instructions on a machine readable medium and/orcomputer-readable medium, which may be incorporated into a computerprogram product.

The preceding description of the disclosed aspects is provided to enableany person skilled in the art to make or use the present invention.Various modifications to these aspects will be readily apparent to thoseskilled in the art, and the generic principles defined herein may beapplied to other aspects without departing from the scope of theinvention. Thus, the present invention is not intended to be limited tothe aspects shown herein but is to be accorded the widest scopeconsistent with the principles and novel features disclosed herein.

What is claimed is:
 1. A surgical system comprising: a robotic armincluding multiple axes between a proximal end and a distal end of therobotic arm, the robotic arm having an outer surface and defining anopening extending through the outer surface; an end effector attached tothe distal end of the robotic arm; and a marker system for tracking therobotic arm using a motion tracking system, the marker systemcomprising: a light source located below the outer surface of therobotic arm and supported within the robotic arm to emit light throughthe opening; and a marker comprising an optical diffuser configured toreleasably attach to the robotic arm above the outer surface of therobotic arm and to optically couple the light source to the opticaldiffuser.
 2. The surgical system of claim 1, wherein power for the lightsource is provided through the robotic arm.
 3. The surgical system ofclaim 1, wherein the optical diffuser comprises a spherical orsemi-spherical outer surface.
 4. The surgical system of claim 1, whereinthe marker system comprises a plurality of markers releasably attachedto the robotic arm.
 5. The surgical system of claim 4, wherein the lightsource further comprises a plurality of light sources located within therobotic arm, and each of the plurality of markers is optically coupledto a respective light source of the plurality of light sources.
 6. Thesurgical system of claim 4, wherein each of the plurality of markers areoptically coupled to the light source located within the robotic arm. 7.The surgical system of claim 1, wherein the marker is releasablyattached to the robotic arm; and wherein a surgical drape is disposedabove the outer surface of the robotic arm and between the robotic armand the optical diffuser of the marker.
 8. The surgical system of claim7, wherein the surgical drape is disposed between the optical diffuserand the light source.
 9. The surgical system of claim 1, furthercomprising an optical guide optically coupled to the light source anddisposed in the opening defined by the outer surface of the robotic arm,the marker configured to releasably attach to the optical guide.
 10. Thesurgical system of claim 9, wherein the optical guide comprises a matingfeature that facilitates releasable attachment of the marker to theoptical guide.
 11. The surgical system of claim 9, wherein a surgicaldrape is disposed between the optical guide and the optical diffuser.12. The surgical system of claim 1, wherein the marker comprises a baseportion that includes an optical guide, the base portion disposed in theopening defined by the outer surface of the robotic arm.
 13. Thesurgical system of claim 12, wherein a surgical drape is disposedbetween the light source and the optical guide.
 14. A surgical systemcomprising: a robotic arm including multiple axes between a proximal endand a distal end of the robotic arm, the robotic arm having an outersurface and defining an opening extending through the outer surface; anend effector attached to the distal end of the robotic arm; a surgicaldrape covering the robotic arm; and a marker system for tracking therobotic arm using a motion tracking system, the marker systemcomprising: a light source located below the outer surface of therobotic arm and supported within the robotic arm to emit light throughthe opening; an optical guide optically coupled to the light source andextending between a first end disposed in the opening defined by theouter surface of the robotic arm and a second end; and a markercomprising an optical diffuser configured to releasably attach to secondend of the optical guide to optically couple the light source to theoptical diffuser, with the surgical drape disposed between the lightsource and the optical diffuser.
 15. The surgical system of claim 14,wherein the surgical drape is disposed between the second end of theoptical guide and the optical diffuser.
 16. The surgical system of claim14, wherein the surgical drape is disposed between the light source andthe first end of the optical guide.
 17. The surgical system of claim 14,wherein power for the light source is provided through the robotic arm.18. The surgical system of claim 14, wherein the optical diffusercomprises a spherical or semi-spherical outer surface.
 19. The surgicalsystem of claim 14, wherein the marker system further comprises aplurality of markers releasably attached to the robotic arm.
 20. Thesurgical system of claim 19, wherein the light source further comprisesa plurality of light sources located within the robotic arm, and each ofthe plurality of markers is optically coupled to a respective lightsource of the plurality of light sources.